1. Field of the Invention
The present invention relates generally to computer graphics, such as: video games, medical imaging, digital watermarking and the like. More particularly, this invention relates to methods for greatly enhancing the variability of computer graphics from limited sources utilizing minimal processing and bandwidth. Specifically, this innovation resolves the problem of the generation of a video file or stream in real time producing complex video output consisting of realistic models of three-dimensional (3D) objects with varying backdrop scenes.
2. Background
Computer generated graphics have substantially progressed since their introduction in the 1950's. One of the first interactive video games to feature recognizable, interactive graphics—“Tennis for Two”—was created for an oscilloscope by William Higinbotham to entertain visitors in 1958 at Brookhaven National Laboratory by simulating a tennis match. In 1961 at MIT (Massachusetts Institute of Technology), Steve Russell, created the second video game, “Spacewar” running on a Digital Equipment Corporation (DEC) PDP-1 model computer. Both of these pioneering video games employed extremely simple line and dot graphic primitives bearing only a faint resemblance to any real world counterparts.
In the 1970's the concepts of the hidden-surface algorithm, Gouraud and Blinn-Phong shading models, Z-axis plotting, bump mapping, and other innovations provided early breakthroughs in computer generated graphics that laid the foundation for much more detailed video graphics. However, the computer processing power available at the time hampered the lifelike appearance of computer video generated by these techniques.
Now, with the emergence of 32-bit and higher microprocessors as well as specialized Graphics Processing Units (GPUs) hardware, pre-rendered computer graphics used in films and video games are nearly scientifically photorealistic—i.e., almost impossible to discern the difference between real world and computer generated. However, real time graphics produced on very high-end systems, while closely simulating photorealism, do not achieve the photorealistic levels of pre-rendered graphics. The difference in output quality being attributable to the requirement for real-time graphics to produce an uninterrupted continuous varying video file or stream.
Thus, being able to generate photorealistic appearing 3D graphics in real time has become the holy grail of the video gaming industry and drives a continued push in hardware and software capability with new innovations that provide a strategic, albeit temporary, advantage for the innovating entity. Known techniques, and traditional approaches take the standard 3D geometry and texture model approach, rendering pixels using light and shading models in real time. Unfortunately, this approach has led to computationally expensive systems requiring dedicated hardware GPUs in conjunction with high-speed servers with sizable communication bandwidth requirements. Such hardware and bandwidth and the lack of it in mobile and home user environments have proven to be problematic.
For example, if photorealistic appearing 3D graphics are rendered at a central site (i.e., “render farms” with significant computational assets running continuously over hours) and downloaded to home and mobile systems in more or less real time the computational requirements problem can be solved by concentrating the expensive graphics hardware at the host's central site and distributing the generated photorealistic looking 3D graphics as video streams to individual consumers with mobile, home, and other platforms. Thus, in this model, the consumer's hardware is principally performing a display function that is not necessarily computationally intensive. However, this model has the disadvantage of high bandwidth requirements on a continuous basis that are typically expensive and may not be available to many users—especially mobile users. Conversely, if photorealistic looking 3D graphics are rendered on the consumer's device, the bandwidth requirements can be reduced; however, typically the consumer's device(s) is not able capable of performing computational processing rendering in real time with the graphic output quality significantly degraded at best.
Another example is for the photorealistic looking 3D graphics to be pre-rendered and downloaded to the consumer's device in advance of interactive play, thereby maintaining high levels of computational processing at a central site while at the same time reducing bandwidth requirements since the graphics could be downloaded in time multiples of their real time playback. However, the typical pre-rendered 3D graphics large file size severely restricts the variety of options available for real time game playback thereby significantly reducing consumer enjoyment of multiple plays since the video will be familiar after only a few viewings.
With medical imaging the tradeoffs between computationally expensive hardware and bandwidth are typically reduced with most viewing local to the hardware that generated the images. However, medical imaging typically is associated with very complex 3D models (i.e., human body) from multiple scan sources where the doctor chooses in real time to display various two dimensional (2D) slices of “compound scans” of the body that were previously acquired. Thus, the presentation of these 2D slices must occur real time or near real time with an accompanying substantial processing burden. When dynamic information (e.g., Doppler blood flow) is superimposed on these 2D slices (e.g., false color velocity of blood flow) the computational requirements become even more extensive. If these types of 2D compound scans are transmitted to a remote location, the available bandwidth also can potentially become problematic ultimately resulting in the sort of tradeoffs previously discussed with gaming.
Finally, pre-rendered graphics digital watermarking is well known in the art and is often utilized to ensure copyright protection as well as embedding other metadata. However, digital watermarking with real time generated graphics is virtually unknown, partially due to the computational difficulties of embedding digital watermarks in video streams generated real time. The computational hardware required to include digital watermarks into a real time video stream being typically prohibitively expensive—both in terms of computational cycles and economics.
Attempts have been made to reduce processing requirements for rendering images in real time. For example, U.S. Pat. No. 5,860,924 (Quistgaard) teaches serially ganging multiplicities of processors each with specific image processing tasks thereby reducing the processing requirements on any single processor. However, “Quistgaard” typically requires expensive custom hardware or at the very least multiplicities of servers in serial and parallel configurations with the associated logistical and economic problems inherent in these types of complex processing configurations.
U.S. Patent Application Publication No. 2008/0012870 (Gies et al.) discloses analyzing each frame of a video stream and related color profile indicating parameters of the color space of the video stream and then converting each frame from a source color space to a working color space. The intent of this process is to convert in real time a video stream from one format to another—e.g., composite NTSC (National Television System Committee) format to HDTV (High-Definition TeleVision) format (paragraph [0024] of “Gies”). Thus, while “Gies” discloses efficiently converting one format of video to another, it does not address the core problems of source computational power and bandwidth tradeoffs nor how to maintain photorealistic looking graphics on consumer devices with variety and dynamic outcomes determined in near real time.
Color space conversion is also taught in U.S. Pat. No. 7,593,021 (Tynefield, Jr. et al.). In this process, color data is converted from one space to another with a driver determining if a color conversion function will be enabled and if so either with a single linear conversion or flagged to perform a non-linear conversion. Again, while this process discloses efficient methods of altering a video stream or file, it does not address the core problems of source computational power and bandwidth tradeoffs nor how to maintain photorealistic looking graphics on consumer devices with variety and dynamic outcomes determined in near real time.
U.S. Pat. No. 7,038,690 (Wilt et al.) discloses methods and systems for interfaces between video applications and consumer display screens that allow applications to intelligently use display resources of the consumer's device with a graphics arbiter providing display environment information to the video application, possibly transforming the video stream or file or allowing another application to transform the video stream or file. With “Wilt” the graphics arbiter also informs applications of the estimated time when the next frame will be displayed on the screen, thereby allowing applications to tailor output to the estimated display time, thus improving output quality while decreasing resource waste by avoiding the production of “extra” frames that will never be viewed. Thus, with “Wilt”, processing efficiencies are gained via the graphics arbiter although source computational power and bandwidth tradeoffs are not significantly reduced. Additionally, “Wilt” makes no attempt to resolve the problem of maintaining photorealistic looking 3D graphics on consumer devices with variety and dynamic outcomes determined in near real time.
Thus, there is a need for a method or system to reduce the tradeoff between source computational power and bandwidth when transmitting a video stream or file of photorealistic appearing graphics from a central site to a consumer device. Ideally, this method or system would also increase the variety of dynamic outcomes displayable in near real time.